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ABSTRACT 

NeMeSys  research  project  aims  at  designing  and  de¬ 
veloping  a  generic  /  interactive  platform  for  building  and 
updating  dynamic  3D  urban  area  models  based  on  struc¬ 
tural  information,  stored  in  hyper-models,  onto  which 
semantic  information  can  be  superimposed.  A  hyper¬ 
model  is  defined  as  a  combination  of  three  models:  geo¬ 
metric,  graphics  rendering  and  behaviour  models.  Three 
fundamental  tasks  are  achieved  at  different  resolution 
levels:  visualization  of  the  entities  (represented  by  hyper¬ 
models)  populating  the  environment,  description  of  the 
geometry  and  semantics  of  the  entities,  and  simulation  of 
phenomena  (e.g.  physics-based)  affecting  these  entities. 

1.  INTRODUCTION 

Trends  in  demographics  indicate  a  continuous  and 
significant  increase  in  both  the  number  and  size  of  Urban 
Areas  (UAs)  throughout  the  world.  An  intricate  topogra¬ 
phy  and  a  larger  concentration  of  population  in  UAs  sug¬ 
gest  that  many  future  urban  operations  should  exploit  a 
new  type  of  terrain  information  for  dealing  with  these 
three-dimensional  environments,  which  are  the  most  dif¬ 
ficult  terrains  in  which  to  conduct  military  operations. 
Canadian  Eorces  must  be  prepared  to  face  the  challenge 
of  conducting  military  operations  in  UAs  (ever  changing 
blends  of  manmade,  natural  and  human  features)  more 
than  ever  before. 

The  availability  of  interactive  /  evolutive  3D  UA 
descriptions  based  on  an  assembly  of  interacting  models 
is  of  great  interest  to  Canadian  Eorces  whose  international 
mandate  has  clearly  shifted  from  conventional  combat 
operations  to  missions  in  UAs  for  which  traditional  2D 
static  information  no  longer  suffice  in  supporting  military 
operations. 

Urban  operations  require  a  thorough  understanding  of 
structural  and  semantic  data  of  the  UA,  (Directorate  of 
Land  Strategic  Concepts,  2003 )\  as  well  as  a  complete 


p.  113,  "'Future  conflicts  will  be  waged  in  complex  terrain, 
particularly  within  an  urban  environment,  most  likely  against  an 
unconventional  or  asymmetric  force.  The  urban  scenario  poses 


and  dynamic  visual  representation  of  these  settings. 

Structural  data  (ground  /  underground  infrastructures, 
urban  topography)  pertains  to  the  shape  of  the  compo¬ 
nents  of  the  UA  and  forms  the  skeleton  onto  which  se¬ 
mantic  data  (critical  assets,  access  issues,  infrastruc¬ 
ture.  . .)  is  superimposed  for  enriching  the  UA  model. 

The  use  and  exploitation  contexts  of  3D  UA  descrip¬ 
tions  include  but  are  not  limited  to:  (1)  synoptic  views  of 
an  UA  (2)  updating  (off-line  or  on-line  -  near  real  time) 
of  models  (updating  can  be  local  or  global),  (3)  visualiza¬ 
tion  /  representation  of  the  state  of  the  situation  at  various 
levels,  (4)  mission  planning  /  monitoring  and  training  for 
that  mission  (including  mission  rehearsal),  (5)  spatial 
analysis,  inter  visibility  and  marksmen's  location,  (6)  battle 
space  analysis  such  as  elements  of  the  intelligence,  etc. 
Each  of  these  contexts  must  rely  on  rich  and  accurate  dy¬ 
namic  descriptions  in  3D.  A  3D  description  adds  a  new 
dimension  to  the  diversity  of  traditional  2D  visual  repre¬ 
sentations  (maps  -  road  maps,  topographic  maps,  aerial 
and  remote  sensing  images,  videos...).  The  creation  of  a 
3D  UA  model  is  constrained  by  the  quality  and  availabil¬ 
ity  of  data,  the  complexity  of  the  urban  terrain,  the  sched¬ 
ule,  the  budget,  the  targeted  purpose,  the  skill  and  soft¬ 
ware. 

Until  now,  UA  descriptions  have  mainly  focused  on 
photorealistic  rendering  (Hu  et  ai,  2003)(Letourneau  et 
al.,  2003)  of  the  components  of  interest  with  little  concern 
for  adding  behaviour  (such  as  physics-based)  to  these 
components.  Based  on  this  analysis,  there  is  an  urgent 
need  for  a  platform  for  supporting  the  construction  and 
exploitation  of  UA  descriptions  in  a  Synthetic  Environ¬ 
ment  (SE)  whose  components  exhibit  dynamic  behaviour. 

This  paper  presents  the  conceptual  view  of  (1)  UAs 
and  (2)  NeMeSys,  a  generic  and  interactive  platform  for 
building,  updating,  visualizing  and  exploiting  rich  and 
accurate  dynamic  3D  UA  descriptions  in  a  SE. 


significant  challenges  for  reconnaissance  and  surveillance,  both 
in  terms  of  the  infrastructure  and  terrain,  and  in  accurately 
assessing  the  nature  of  the  threat.  [...]  Indeed,  much  more  detail 
is  required  on  the  environment  itself,  on  critical  nodes  (e.g. 
hospitals,  water  plants,  power  grids)  and  on  inhabitants  in  gen¬ 
era!' 
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2.  URBAN  AREA 

UAs  are  complex,  dynamic  environments.  Each  one 
is  distinct  and  will  react  to  and  affect  military  operations 
differently.  However,  all  UAs  share  three  distinguishing 
characteristics,  also  called  an  “urban  triad”,  that  are  gen¬ 
erally  so  intertwined  as  to  be  virtually  inseparable  (see 
Fig.  1).  The  first  characteristic  is  a  complex  manmade 
physical  terrain  superimposed  on  an  existing  (complex) 
natural  terrain.  This  terrain  consists  of  structures  and  fa¬ 
cilities  of  varying  types,  sizes,  materials,  and  construc¬ 
tions  arranged  either  orderly  or  randomly.  A  population 
defined  by  a  set  of  demographic  considerations  is  the  sec¬ 
ond  important  characteristic  of  UAs  (see  Fig.  2).  This 
population  inhabits,  works  in,  and  uses  the  manmade  and 
natural  terrain.  Socio-cultural  considerations  also  help  in 
describing  the  population.  The  third  characteristic  that 
needs  to  be  considered  is  the  physical  /  service  infrastruc¬ 
ture  upon  which  the  area  depends  (see  Fig.  3).  This  infra¬ 
structure  that  links  urban  terrain  and  population  may  also 
occupy  manmade  physical  terrain  and  provides,  among 
other  things,  human  services  and  cultural  and  political 
structure  for  the  UA. 
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Two  types  of  input  data  need  to  be  considered  in 
building  3D  UA  descriptions:  structural  and  semantic 
data.  Structural  data,  which  is  closely  related  to  urban 


terrain,  present  a  mix  of  horizontal,  vertical,  interior,  exte¬ 
rior  and  subterranean  forms  (see  Fig.  4,  Fig.  5)  superim¬ 
posed  on  the  natural  relief,  drainage,  and  vegetation  (see 
Fig.  6).  Semantic  data  are  linked  to  every  UA  characteris¬ 
tic  (urban  terrain,  society  and  infrastructure)  but  in  a  dif¬ 
ferent  way.  Fig.  7  and  Fig.  8  show  different  views  of  the 
issues  that  semantic  data  could  deal  with. 


Fig.  4  Urban  terrain  -  Conceptual  view  of 
the  manmade  terrain 
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Fig.  7  Semantic  data:  Conceptual  view  1 


3.  NEMESYS  PLATFORM 

The  use  of  SEs  is  becoming  a  practical  and  economi¬ 
cal  approach  for  military  operations.  Flexible  and  efficient 
platforms,  such  as  NeMeSys,  are  needed  for  constructing, 
updating,  and  exploiting  these  SEs  not  only  for  visualiza¬ 
tion  purposes  but  also  for  simulating  behavioural  phe¬ 
nomena.  NeMeSys  platform  explores  completely  new 
ways  of  constructing  3D  UA  models  that  can  not  only  be 
updated  dynamically  but  that  can  also  adopt  realistic  be¬ 
haviour  in  a  simulation  context.  The  structure  of  the  mod¬ 
els  is  such  that  structural  and  semantic  information  can  be 
easily  made  available  to  users. 

Such  a  platform  integrating  UA  visualization,  geo¬ 
metric  and  behaviour  descriptions  has  a  large  potential  for 
innovation  both  from  the  conceptual  and  the  system 


design  standpoints.  Although  significant  progress  has 
been  made  in  past  years  with  respect  to  model  visualiza¬ 
tion  and  graphics  rendering,  namely  due  to  the  progress 
made  in  the  development  of  physics-based  photometric 
models  and  image-based  rendering,  very  few  platforms 
address  the  problem  of  integrating  and  supporting  visuali¬ 
zation,  geometric  and  behaviour  descriptions  in  a  same 
paradigm  (Burdea  and  Coiffet,  2003). 

Fig.  9  shows  characteristics  to  consider  (a  part)  for 
NeMeSys  platform.  One  of  the  most  important  is  the  re¬ 
quirement  at  the  origin  of  the  3D  UA  description  because 
it  influences  all  other  characteristics. 

3.1  NeMeSys  Architecture 

NeMeSys  platform  architecture  is  shown  in  Fig.  10. 
The  platform  accepts  various  types  of  input  data  -  either 
structural  or  semantic,  builds  hyper-models  for  describing 
the  geometry,  appearance  and  behaviour  of  the  entities  in 
the  SE,  and  allows  users  to  interact  with  the  entities 
through  a  human-machine  interface.  The  UA  description 
is  an  assembly  of  interacting  hyper-models.  A  hyper¬ 
model  is  defined  as  a  model  combining  geometric  (shape  / 
structure),  photometric  (appearance),  and  physical  /  se¬ 
mantic  (behaviour  /  purpose)  descriptions  of  the  entities 
composing  the  SE.  Since  building  such  hyper-models  is  a 
complex  task,  the  platform  combines  software  modules 


for  automatic  model  construction  as  well  as  human-in- 
the-loop  interfaces  for  refining  the  models. 

Contrary  to  most  current  platforms  which  focus  on 
rendering  photorealistic  models  of  UAs  but  have  no  pro¬ 
visions  for  updating  the  models,  NeMeSys  puts  the  em¬ 
phasis  on  embedding  behaviour  (Bernier  et  al,  2002)  into 
the  models  and  on  the  dynamic  updating  (Blais  et  al, 
2003)  of  both  geometry  and  appearance.  In  addition  to 
behaviour,  NeMeSys  aims  at  providing  users  with  com¬ 
plementary  semantic  information  on  the  models  either  for 
visualization  purposes  or  for  enriching  simulations  (see 
Fig.  11  that  shows  roughly  the  interaction  between  the 
structural  and  semantic  layers).  In  NeMeSys,  behaviour  is 
controlling  geometry  and  appearance  in  a  dynamic  way, 
as  it  should  be  in  a  realistic  situation. 

Adapted  data  structures 

A  key  element  of  NeMeSys  platform  is  that  it  does 
not  impose  a  single  data  structure  onto  which  models  of 
the  hyper-model  must  be  implemented  but  rather  attempts 
to  exploit  standard  data  structures  that  are  cleverly  inter¬ 
faced  with  each  other.  This  is  clearly  visible  in  Fig.  10:  a 
hyper-model  can  exploit  a  first  data  structure  for  graphics 
rendering,  a  second  data  structure  for  describing  the  ge¬ 
ometry  of  an  entity,  and  a  third  data  structure  for  simula¬ 
tion  of  its  behaviour.  In  addition,  semantic  information  on 
an  entity  (a  set  of  entities)  can  be  overlaid  on  its  (their) 
hyper-model(s)  seamlessly. 

Modular  architecture 

The  main  advantage  of  NeMeSys  is  that  it  eliminates 
the  coupling  between  the  three  models  describing  the  hy¬ 
per-model  for  adopting  a  modular  architecture  where 
these  three  elements  interact  with  each  other  through 
cleverly  designed  interfaces  instead  of  being  intertwined 
in  a  common  model  that  is  not  only  difficult  to  build  but 
also  to  update.  Duplicating  models  for  describing  differ¬ 
ent  aspects  of  the  same  entity  may  first  appear  as  a  weak¬ 
ness.  However,  a  careful  analysis  of  the  problem  demon¬ 
strates  that  adopting  this  strategy  shows  a  significant  ad¬ 
vantage  over  existing  approaches  based  on  a  single  data 
structure. 

First,  using  multiple  complementary  representations 
of  an  entity  allows  each  representation  to  achieve  the 
functionality  it  has  been  designed  and  optimized  for.  In 
NeMeSys  for  instance,  the  structure  of  standard  formats 
for  describing  geometry  is  studied  carefully  to  check 
whether  they  can  be  interfaced  seamlessly  with  standard 
formats  for  graphics  rendering  such  as  those  found  in 
widely  used  graphics  engines.  More  importantly,  NeMe¬ 
Sys  investigates  ways  to  add  behaviour  (e.g.  physics- 
based)  to  the  geometry  so  that  shape  becomes  an  effect  of 
behaviour  instead  of  being  a  cause. 


Fig.  9  NeMeSys  platform:  characteristics  to  consider 


Secondly,  the  hyper-model  approach  maintains  the 
independence  between  tasks  that  were  not  planned  to  be 
combined  in  the  first  place.  For  instance,  graphics  render¬ 
ing  and  simulation  of  a  behaviour  rarely  need  to  run  at  the 
same  rate  and,  using  separate,  yet  coordinated  models  for 
each  purpose  achieves  better  optimized  results.  Our  claim 
is  that  it  is  the  simulation  module  controlling  behaviour 
that  must  also  control  changes  in  geometry  and  appear¬ 
ance.  Most  current  approaches  let  graphics  rendering  con¬ 
trol  behaviour,  which  results  in  computational  bottlenecks 
caused  by  the  intrinsic  nature  of  graphics  rendering  data 
structures.  Research  work  is  thus  focussing  on  defining  a 
paradigm  through  which  simulation  controls  geometry 
and  appearance  (graphics  rendering). 

Thirdly,  adopting  standard  data  representations  spe¬ 
cifically  designed  for  visualization,  geometric  description, 
and  simulation  increases  the  portability  and  reusability  of 
the  hyper-models  as  well  as  their  exploitation  in  a  net¬ 
worked  distributed  environment.  Even  more  importantly, 
this  strategy  removes  the  complexity  from  the  algorithms 
and  transposes  it  to  a  data  structure  that  is  rich  enough  to 
support  such  complexity.  Finally,  using  standard  repre¬ 
sentations  supports  hyper-model  interoperability,  portabil¬ 
ity,  evolution  and  maintenance  and  increases  their  com¬ 
patibility  with  other  existing  tools  and  commercial  prod¬ 
ucts. 

System  development 

From  the  standpoint  of  system  development,  NeMe- 


Sys  platform  architecture  also  presents  significant  advan¬ 
tages.  First,  it  allows  concurrent  development  of  the  three 
models  (structural,  visual  and  behavioural).  Secondly,  it 
allows  to  clearly  define  the  interfaces  between  the  models 
and  to  better  plan  system  integration.  Finally,  system  tests 
can  also  be  planned  more  easily  since  sub-systems  can  be 
tested  independently  and  then  be  tested  as  a  whole  system 
following  integration. 

Behaviour-centric  platform 

NeMeSys  is  behaviour-centric  and  considers  geome¬ 
try  and  appearance  as  clients  of  entities’  behaviour.  The 
benefits  of  adopting  this  paradigm,  non  widespread  in  the 
community  (Burdea  and  Coiffet,  2003),  is  that  geometry 
and  appearance  can  change  and  be  updated  at  their  own 
rate  under  the  control  of  behaviour  as  objects  do  in  the 
real  world  (e.g.  aging  of  a  building  is  not  a  consequence 
of  its  appearance  changing  with  time  but  a  change  in  ap¬ 
pearance  is  rather  the  result  of  aging).  The  hyper-model 
approach  improves  the  integration  of  many  essentials 
components  of  SE  to  support  R&D  activities  and  to  put 
innovation  into  an  urban  operational  context.  Then,  sys¬ 
tems  and  their  components  could  be  placed  simultane¬ 
ously  into  a  common  environment,  validating  their  inter¬ 
operability  and  their  integration  with  their  context  of  op¬ 
eration,  both  from  a  technology  point  of  view  and  from  a 
doctrine  and  techniques,  tactics  and  procedures  point  of 
view. 
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Fig.  10  NeMeSys  platform  architecture 


structural  Data  Dissemination 


Fig.  11  Interaction  between  structural  and  semantic  data  layers 


Functionalities 

With  the  NeMeSys  platform,  the  Canadian  Forces 
will  be  in  a  position  to:  (1)  build  complex  multi-scale 
models  of  UAs  (Borgeat  et  al,  2003),  (2)  update  these 
models  dynamically,  (3)  visualize  the  models  in  3D  from 
different  vantage  points  (Laurendeau  et  al,  2003)  (Mok- 
htari  et  al.,  2004a)  combining  different  human  points  of 
view,  (4)  embed  behaviour  and  functionality  on  the  mod¬ 
els  for  simulation,  and  (5)  interact  with  the  models  in  a  SE 
(Laurendeau  et  al,  2003)(Mokhtari  et  ah,  2004b). 

CONCLUSION 

As  NeMeSys  aims  at  providing  the  next  generation  of 
3D  UA  models,  it  is  well  adapted  to  the  future  operations 
of  Canadian  Forces  that  will  require  the  ability  to  plan, 
monitor,  train  and  rehearse  in  a  virtual,  interoperable  en¬ 
vironment.  Furthermore,  NeMeSys  could  offer,  in  a  near 
future,  dynamic  training  capabilities  in  a  complex  envi¬ 
ronment  by  combining  simulations,  combat  system  mod¬ 
els,  and  locomotion  interface  in  virtual  scenarios.  This 
research  work  also  aims  to  contribute  to  produce  more 
realistic  synthetic  urban  environments  for  many  R&D 
activities  which  will  benefit  not  only  from  a  good  geomet¬ 
rical  representation  of  the  environment  but  also  from  a 
good  behavioural  representation  of  the  reality.  The  robust 
and  open  approach  will  be  more  resilient  to  change,  im¬ 
proving  the  upgrade  and  the  maintenance  of  such  envi¬ 
ronments  for  R&D  activities. 
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